Method and electronic device for motion prediction

ABSTRACT

A method for motion prediction includes receiving spatial information output by a radio-wave sensor, wherein the spatial information includes position and velocity of at least one point; receiving an image captured by a camera; tracking at least one object based on the spatial information and the image to obtain a consolidated tracking result; predicting a motion trajectory of the at least one object based on the consolidated tracking result to obtain a prediction result; and controlling the camera according to the prediction result.

CROSS REFERENCE TO RELATED APPLICATIONS Field of the Disclosure

The present invention is related to a method and an electronic deviceapplying the method, and in particular it is related to a method and anelectronic device using a depth sensor and a camera for motionprediction.

DESCRIPTION OF THE RELATED ART

When a person wants to take a photograph or a video with a camera, it isoften too late to capture the key moment. When the user snaps thephotograph early to capture a series of images, the snapshots may stillmiss the key moment, due to the camera having limited frames per second(fps). When utilizing image-based algorithms, such as machine learningfor classification and recognition to assist object tracking foradaptive snapshots, numerous challenges result in inadequate image/videoquality and poor user experience. For example, the challenges may betime lag and high power consumption, hijacking, centralization,drifting, and loss of focus.

Electronic Image Stabilization (EIS) is based on image content detectionto track and compensate the camera movement. Similar to theaforementioned imaged-based object tracking (although with differentalgorithms), there are still challenges that can result in imperfect EISoutcomes. For example, the challenges may be hijacking, time lag, highsystem power consumption, and loss of effective Field of View (FoV).

BRIEF SUMMARY OF THE DISCLOSURE

In order to resolve the issue described above, the present inventionprovides a method for motion prediction. The method includes: receivingspatial information output by a radio-wave sensor, wherein the spatialinformation includes position and velocity of at least one point;receiving an image captured by a camera; tracking at least one objectbased on the spatial information and the image to obtain a trackingresult; predicting a motion trajectory of the at least one object basedon the tracking result to obtain a prediction result; and controllingthe camera according to the prediction result.

According to the method described above, the spatial information furtherincludes a frame-level confidence indicator.

According to the method described above, the frame-level confidenceindicator is determined based on a stability of a signal power level ofa reflected wave received by the radio-wave sensor within a frame.

According to the method described above, the step of tracking the atleast one object based on the spatial information and the image toobtain the tracking result, includes: processing the image to obtain atleast one property of the at least one object in the image; and fusingthe spatial information with the at least one property of the at leastone object in the image to obtain the tracking result.

According to the method described above, the at least one property ofthe at least one object in the image is obtained by processing the imageusing at least one of Artificial Intelligence (AI), Machine Learning(ML) content detection and Computer Vision (CV).

According to the method described above, the step of controlling thecamera according to the prediction result, includes: setting at leastone of camera parameters and image parameters based on the predictionresult; and capturing a final image based on the set camera parametersand the set image parameters.

According to the method described above, the camera parameters includeat least one of a shutter timing, a shutter speed and a focal length ofthe camera based on the prediction result, and the image parameters arededicated for image processing.

According to the method described above, the step of tracking the atleast one object based on the spatial information and the image toobtain the tracking result, includes generating a first tracking resultbased on the spatial information; generating a second tracking resultbased on the image; and unifying the first tracking result and thesecond tracking result to obtain the consolidated tracking result.

According to the method described above, the step of unifying the firsttracking result and the second tracking result, includes: setting theweights of the first tracking result and the second tracking resultbased on at least one of the spatial information and the image; andselecting one of the first and second tracking results with the higherweight as the consolidated tracking result.

The present invention also provides a method for Electronic ImageStabilization (EIS), includes: receiving spatial information output by aradio-wave sensor, wherein the spatial information includes position andvelocity of at least one point; receiving an image captured by a camera;determining, based on the spatial information and the image, that motionof the at least one object is local motion if the motion of the at leastone object is inconsistent with most other objects; determining, basedon the spatial information and the image, that the motion of the atleast one object is global motion if the motion of the at least oneobject is consistent with most other objects; and filtering the globalmotion for EIS cropping.

The present invention also provides an electronic device. The electronicdevice includes a radio-wave sensor, a camera, and a processor. Theradio-wave sensor is configured to provide spatial information includingposition and velocity of at least one point. The camera is configured toprovide an image. The processor is configured to execute the followingsteps: receiving the spatial information including the position andvelocity of the at least one point; receiving the image captured by thecamera; tracking at least one object based on the spatial informationand the image to obtain a tracking result; predicting a motiontrajectory of the at least one object based on the tracking result toobtain a prediction result; and controlling the camera according to theprediction result.

According to the electronic device above, the spatial informationfurther includes a frame-level confidence indicator; the frame-levelconfidence indicator is determined based on a stability of a signalpower level of a reflected wave received by the radio-wave sensor withina frame.

According to the electronic device above, the processor is configured totrack the at least one object based on the spatial information and theimage to obtain the tracking result, including: processing the image toobtain at least one property of the at least one object in the image;and fusing the spatial information with the at least one property of theat least one object in the image to obtain the tracking result.

According to the electronic device above, the at least one property ofthe at least one object in the image is obtained by processing the imageusing at least one of Artificial Intelligence (AI), Machine Learning(ML) content detection and Computer Vision (CV).

According to the electronic device above, the processor is configured totrack the at least one object based on the spatial information and theimage to obtain the tracking result, including: generating a firsttracking result based on the spatial information; generating a secondtracking result based on the image; and unifying the first trackingresult and the second tracking result to obtain the consolidatedtracking result.

According to the electronic device above, the processor is configured tounify the first tracking result and the second tracking result to obtainthe tracking result, including: setting weights of the first trackingresult and the second tracking result based on at least one of thespatial information and the image; and selecting one of the first andsecond tracking results with the higher weight as the consolidatedtracking result.

According to the electronic device above, the processor is configured tocontrol the camera according to the prediction result, including:setting at least one of camera parameters and image parameters based onthe prediction result; and capturing a final image based on the setcamera parameters and the set image parameters.

According to the electronic device above, the camera parameters includeat least one of a shutter timing, a shutter speed and a focal length ofthe camera based on the prediction result, and the image parameters arededicated for image processing.

The present invention also provides an electronic device for EIScropping. The electronic device includes a radio-wave sensor, a cameraand a processor. The radio-wave sensor is configured to provide spatialinformation including position and velocity of at least one point. Thecamera is configured to provide an image. The processor is configured toexecute the following steps: receiving spatial information output by theradio-wave sensor; receiving the image captured by the camera;determining, based on the spatial information and the image, that motionof at least one object is local motion if the motion of the at least oneobject is inconsistent with most other objects; determining, based onthe spatial information and the image, that the motion of the at leastone object is global motion if the motion of the at least one object isconsistent with most other objects; and filtering the global motion forEIS cropping.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure can be more fully understood by reading the subsequentdetailed description with references made to the accompanying figures.It should be understood that the figures are not drawn to scale inaccordance with standard practice in the industry. In fact, it isallowed to arbitrarily enlarge or reduce the size of components forclear illustration. This means that many special details, relationshipsand methods are disclosed to provide a complete understanding of thedisclosure.

FIG. 1 is a flow chart of a method for motion prediction in accordancewith some embodiments of the present invention.

FIG. 2 is a schematic diagram of a first application of the method inFIG. 1 in accordance with some embodiments of the present invention.

FIG. 3A is a flow chart of a method of the first application in FIG. 2in accordance with some embodiments of the present invention.

FIG. 3B is a flow chart of the other method of the first application inFIG. 2 in accordance with some embodiments of the present invention.

FIG. 4 is a schematic diagram of a second application of the method inFIG. 1 in accordance with some embodiments of the present invention.

FIG. 5 is a schematic diagram of an electronic device 600 in accordancewith some embodiments of the present invention.

FIG. 6 is a flow chart of a method for EIS cropping in accordance withsome embodiments of the present invention.

DETAILED DESCRIPTION OF THE DISCLOSURE

Certain words are used to refer to specific elements in thespecification and the claims. Those with ordinary knowledge in thetechnical field should understand that hardware manufacturers may usedifferent terms to refer to the same component. The specification andthe claims of the present invention do not use differences in names as away to distinguish elements, but use differences in functions ofelements as a criterion for distinguishing. The “comprise” and “include”mentioned in the entire specification and the claims are open-endedterms, so they should be interpreted as “including but not limited to”.“Generally” means that within an acceptable error range, a person withordinary knowledge in the technical field can solve the technicalproblem within a certain error range, and basically achieve thetechnical effect. In addition, the term “coupled” herein includes anydirect and indirect electrical connection means. Therefore, if it isdescribed in the text that a first device is coupled to a second device,it means that the first device can be directly electrically connected tothe second device, or indirectly electrically connected to the seconddevice through other devices or connecting means. The followingdescriptions are preferred ways to implement the present invention. Thepurpose is to illustrate the spirit of the present invention and not tolimit the scope of protection of the present invention.

The following description is the best embodiment expected of the presentinvention. These descriptions are used to illustrate the generalprinciples of the present invention and should not be used to limit thepresent invention. The protection scope of the present invention shouldbe determined on the basis of referring to the scope of the claims ofthe present invention.

FIG. 1 is a flow chart of a method for motion prediction in accordancewith some embodiments of the present invention. As shown in FIG. 1 , themethod for motion prediction of the present invention includes:receiving spatial information output by a radio-wave sensor, wherein thespatial information includes position and velocity of at least one point(step S100); receiving an image captured by a camera (step S102);tracking at least one object based on the spatial information and theimage to obtain a tracking result of each object (step S104); predictinga motion trajectory of each object based on the tracking result toobtain a prediction result of each object (step S106); and controllingthe camera according to the prediction result (step S108).

In step S100, the position of the object may be formed by Cartesiancoordinates, such as (X, Y, Z), or spherical coordinates, such as (r, θ,ψ), but the present invention is not limited thereto. In someembodiments, the position and velocity of the object is sent from aradio-wave sensor. The radio-wave sensor may be a radar, especially amillimeter wave radar. The radio-wave sensor may be any types of radar,for example, a pulse radar, a chirp radar, a frequency modulatedcontinuous wave radar, but the present invention is not limited thereto.In some embodiments, the spatial information further includes aframe-level confidence indicator. The frame-level confidence indicatoris determined based on the stability of the signal power level of thereflected wave received by the radio-wave sensor within a frame. In someembodiments, the radio-wave senor of the present invention transmits 32chirps in each frame. That is, the radio-wave senor calculates theframe-level confidence indicator in each frame based on the power of thereflected wave in each frame. The higher the frame-level confidenceindicator is, the more accurate the position or velocity of the objectis.

Specifically, in some embodiments of step S104, the method for motionprediction includes processing the image to obtain at least one propertyof the object in the image; and fusing the spatial information with theat least one property of the object in the image processing to obtainthe tracking result. In some embodiments, the spatial information isformed by a 4-dimensional point cloud, but the present invention is notlimited thereto. In some embodiments, the term “point cloud” is ageneral term. It can mean one signal point, or a group of points. Thekey and meaning about tracking is across multiple frames. For example,an object is initially located at a first location at a first timepoint, and moves to a second location at a second time point. Theradio-wave sensor detects the object at the first time point to generatea first frame including a group of point cloud at the first location.After that, the radio-wave sensor detects the object at the second timepoint to generate a second frame including another group of point cloudat the second location. The tracking of the object can be done bycalculating moving trajectory of these two groups of point cloud basedon the first frame and the second frame (that is across multipleframes). Typically, each point being tracked has a unique ID. One cantrack a signal point (e.g. ID1) across frames, or multiple pointsindividually and simultaneously (e.g. ID1, ID2, . . . , ID10) acrossframes. In some embodiments, the 4-dimensional point cloud isconstructed by the points representing the position (X, Y, Z) and thevelocity of the object. If the spatial information further includes theframe-level confidence indicator, a 5-dimension point cloud isconstructed accordingly. The frame-level confidence indicator may removethe detected objects with low stability of presence. In someembodiments, the method of the present invention which can generatereal-time 4D/5D point clouds within the FoV in one frame at a higherframes per second (fps) than that from a typical image sensor. Thepenetration capability of radio-wave through non-metallic materialsupports identification of (totally or partially) occluded objects inthe 3D space resulting in early prediction.

In step S104, the method for motion prediction uses a trackingalgorithm, such as an enhanced Kalman filter with vectored derivativesof the radio-wave sensor to track the object. Processing, by the methodof present invention, the 4D/5D point clouds such as the derivative ofvelocity, windowed filtering, tracking, etc. over time plus signal powerand spectral shape to further form up to 7D point cloud may yieldtransition and trajectory of motion indicating intention of objectmovement over space and time. In some embodiments, the at least oneproperty of the object in the image is obtained by processing the imageusing at least one of Artificial Intelligence (AI), Machine Learning(ML) and Computer Vision (CV). In some embodiments, the at least oneproperty of the object in the image may be a scene, a human, a car . . .etc., but the present invention is not limited thereto. In someembodiments, AI (Deep Neural Network: DNN with cognition) is used forscene analysis and semantic segmentation, ML is used for objectclassification and identification such as recognizing a person, and CVis used for object detection such as detecting a human in a boxed area.In some embodiments, AI and ML are exchangeable.

In some embodiments of step S104, the method for motion predictionincludes generating a first tracking result based on the spatialinformation; generating a second tracking result based on the image; andunifying the first tracking result and the second tracking result toobtain the consolidated tracking result. In some embodiments, the stepof unifying the first tracking result and the second tracking result toobtain the consolidated tracking result includes setting weights of thefirst tracking result and the second tracking result based on at leastone of the spatial information and the image; and selecting one of thefirst and second tracking results with the higher weight as theconsolidated tracking result. In some embodiments, the weights of thefirst tracking result and the second tracking result can be initiallyset as the same. The weights of the first tracking result and the secondtracking result can then be adjusted based on at least one of thespatial information and the image. For example, if the acceleration(derivative of velocity) information from an image is ambiguous orlarger than a threshold which is practically impossible in physics, theweight for camera will be decreased. That is, the weight for radio-wavesensor tracking will be increased. Similarly, when a very low lightcondition is detected exceeding the dynamic range of the image signalprocessing, radio-wave sensor may be weighted higher. On the other hand,in the case that there is a huge point cloud from the radio-wave sensorand the present invention wants to track one object with a differentcolor in the image, the camera may be weighted higher. In someembodiments, the method of the present invention sets the weights of thefirst tracking result and the second tracking result based on physics orquality, but the present invention is not limited thereto.

In some embodiments, the method of the present invention furtherincludes a step of selecting the at least one object based on an inputsignal of an user interface before step S104, or a step of selecting theat least one object based on at least parts of the spatial information(for example, positon, velocity or acceleration) of the at least oneobject after step S104.

In step S106, the prediction result is used to obtain at least one ofcamera parameters and image parameters. In some embodiments of stepS108, the method for motion prediction includes setting at least one ofthe camera parameters and the image parameters based on the predictionresult; and capturing a final image based on the set camera parametersand the set image parameters. In some embodiments, the camera parametersinclude at least one of a shutter timing, a shutter speed, and a focallength of the camera. In some embodiments, the image parameters arededicated for image processing. For example, the image parametersincludes the coordinates, the depth distance, the velocity, and theacceleration of the at least one object. The method for motionprediction of the present invention determines the zoom-in, zoom-out,and focus position of the at least one object according to the imageparameters. In some embodiments, if the method of present inventionwould like to capture an image of the moment of jumping of the at leastone object, the method of present invention estimates when to capturethe image at the highest position, to adjust the time to capture theimage, and to adjust the size and the position of the image according tothe temporal information (and/or spatial information) of the at leastone object in the image.

FIG. 2 is a schematic diagram of a first application of the method inFIG. 1 in accordance with some embodiments of the present invention. Asshown in FIG. 2 , an object 200 and an object 202 run to each other faceby face. For example, the object 200 runs from the left to the right,and the object 202 runs from the right to the left. In some embodiments,the image with the objects 200 and 202 on the left of FIG. 2 is capturedby a camera. Since the method for motion prediction of the presentinvention is able to track the object 200 based on the image captured bythe camera, there is a mark frame 210 marked on the object 200, and themark frame 210 may be generated by Artificial Intelligence (AI), MachineLearning (ML) content detection or Computer Vision (CV). In someembodiments in FIG. 2 , point cloud 200′ on the right in FIG. 2represents the motion trajectory of the object 200 detected by aradio-wave sensor, and point cloud 202′ represents the motion trajectoryof the object 202 detected by the radio-wave sensor. In someembodiments, the point clouds 200′ and 202′ are cross-frame pointclouds.

When the object 200 and the object 202 overlap in the image captured bythe camera, that is, the objects 200 and 202 are totally occludedobjects or partially occluded objects, the AI, ML or CV may not identifythe object 200 and the object 202 at the same time, that is, theimage-based detection has failed. Thus, the weight of the image-trackingresult may be decreased. However, the recognition between the object 200and the object 202 is still available and valid based on the pointclouds 200′ and 202′. Therefore, the method of present invention canstill correctly track the object 200 according to the point cloud 200′.

FIG. 3A is a flow chart of a method of the first application in FIG. 2in accordance with some embodiments of the present invention. As shownin step S300, the method for motion prediction of the present inventionfirst previews the spatial information (such as positions, velocities,and frame-level confidence indicators) received from the radio-wavesensor and the image received from the camera. After that, in step S302,the method of the present invention selects the object 200 by markingthe mark frame 210. In step S304, the method of the present inventionprocesses the image to obtain at least one property of the object 200 inthe mark frame 210 of the image. In detail, the method of the presentinvention constructs a 4 or higher-dimensional point cloud (for example,the point clouds 200′ and 202′) based on the position, velocity andadditional properties, such as acceleration, of the objects 200 and/or202. In step S306, the method of the present invention fuses spatialinformation with the at least one property of the object 200 in theimage. In detail, the method of the present invention fuses thehigh-dimensional point cloud with the image processing result (forexample, the mark frame 210 in the image captured by the camera). Thatis, the method of present invention takes the high-dimensional pointcloud and the image processing result into account at the same time forobject tracking. In step S308, the method of the present inventiontracks the object 200 to obtain a tracking result according to bothpoint clouds 200′ and 202′ and the mark frame 210. In detail, the methodof present invention tracks each object (such as the objects 200 and202) in the image to obtain the tracking result of each object accordingto the mark frame 210 and the point clouds 200′ and 202. In someembodiments, the method of the present invention fuses spatialinformation with the at least one property of the object 200 in theimage based on weights between spatial information and the at least oneproperty of the object 200 in the image. For example, at a low lightcondition where camera images are impaired, the spatial informationdetected by the radio-wave sensor would get higher weights. On the otherhand, if the at least one property of the object 200 in the image isstrongly associated with colors and contents that the radio-wave sensorhas no ability to detect such information, the image tracking resultgets higher weights. If the boundaries of physical object have unnaturalmovement, the spatial information detected by the radio-wave sensorwould get higher weights. The unnatural movement may include, forexample, a sudden change in moving directions, which is more often tohappen in the image. In some embodiments, the method of the presentinvention fusing the high-dimensional point cloud with camera, and AIalgorithm can further enhance the performance and recognition rate ofobject detection and tracking. In addition, signal power and spectralshape detected by the radio-wave sensor can be used to (partially)identify and distinguish among objects or among different material toenhance accurate tracking. In some embodiments, the method of thepresent invention also executes coordinates alignment between spatialinformation from the radio-wave sensor and image-based detection.

In step S310, the method of the present invention predicts a motiontrajectory of the object 200 based on the tracking result to obtain aprediction result. In step S312, the method of the present inventionsets at least one of camera parameters and image parameters based on theprediction result. In some embodiments, the camera parameters include atleast one of a shutter timing, a shutter speed, and a focal length ofthe camera. In some embodiments, the image parameters are dedicated forimage processing. In some embodiments, the image parameters can be usedfor camera photographing and object focusing, which are capable ofcapturing the object all the time for subsequent processing. Forexample, the method of present invention adjusts the position and sizeof the image captured by the camera according to the information aboutthe depths and the directions of the two objects in the image, to avoidtracking the wrong object among the two objects. Then, in step S314, themethod of the present invention captures a final image based on thesetting of the camera parameters and the image parameters. Finally, instep S316, the method of the present invention outputs the final image.

FIG. 3B is a flow chart of the other method of the first application inFIG. 2 in accordance with some embodiments of the present invention. Asshown in step S300′, the method for motion prediction of the presentinvention first previews the spatial information (such as positions,velocities, and frame-level confidence indicators) received from theradio-wave sensor and the image received from the camera. After that, instep S302′, the method of the present invention selects the object 200by marking the mark frame 210. In step S306′, the method of the presentinvention generates a first tracking result based on the spatialinformation. In detail, the method of the present invention constructs a4 or higher-dimensional point cloud (for example, the point clouds 200′and 202′) based on the position, velocity and other properties of theobjects 200 and/or 202, and tracks the object 200 to obtain the firsttracking result. At the same time, in step S304′, the method of thepresent invention generates a second tracking result based on the image.In detail, the method of the present invention determines an imageprocessing result based on the image (for example, the mark frame 210)and tracks the object 200 based on the image processing result to obtainthe second tracking result. Furthermore, in step 308′, the method of thepresent invention unifies the first tracking result and the secondtracking result to obtain the consolidated tracking result.

In step 308′, the method of the present invention sets weights of thefirst tracking result and the second tracking result based on at leastone of the spatial information and the image; and selects one of thefirst tracking result and second tracking result with the higher weightas the consolidated tracking result. For example, if the first trackingresult is weighted higher than the second tracking result because thelatter doesn't follow the common physics, the first tracking result isselected. If the weight of the second tracking result is higher becauseof, for example, the additional image content recognition, it isselected.

In step S310′, the method of the present invention predicts a motiontrajectory of the object 200 based on the tracking result to obtain aprediction result. In step S312′, the method of the present inventionsets at least one of camera parameters and image parameters based on theprediction result. In some embodiments, the camera parameters include atleast one of a shutter timing, a shutter speed, and a focal length ofthe camera based on the prediction result. In some embodiments, theimage parameters are dedicated for image processing. Then, in stepS314′, the method of the present invention captures a final image usingthe camera based on the setting of the camera parameters and the imageparameters. Finally, in step S316′, the method of the present inventionoutputs the final image.

FIG. 4 is a schematic diagram of a second application of the method inFIG. 1 in accordance with some embodiments of the present invention. Asshown in FIG. 4 , an object 400 runs on the ground from the left to theright and jumps up and down. In some embodiments, the image with theobject 400 on the left of FIG. 4 is captured by a camera. The image isidentified by Artificial Intelligence (AI), Machine Learning (ML)content detection, or computer vision (CV). In some embodiments, pointcloud 400′ on the right in FIG. 4 represents the motion trajectory ofthe object 400 detected by a radio-wave sensor. Since AI, ML contentdetection or CV is relatively slow and cannot accurately detect thehighest position of the object 400 alone during the jumping, the pointcloud 400′ based on the spatial information from the radio-wave sensorcan assist to detect or predict the highest position of the object 400,so that an image 410 with the object 400 at the highest position whenjumping can be captured by the camera. The flow chart of the method ofthe second application in FIG. 4 is the same as the flow chartsdescribed in FIG. 3A and FIG. 3B, therefore the present invention maynot describe again.

FIG. 5 is a schematic diagram of an electronic device 600 in accordancewith some embodiments of the present invention. The electronic device600 may be a smart phone, a laptop, and a tablet, but the presentinvention is not limited thereto. The electronic device 600 includes aradio-wave sensor 602, a camera 604, and a processor 606. In someembodiments, the radio-wave sensor 602 provides spatial information 610including position and velocity of at least one point. The spatialinformation further includes a frame-level confidence indicator. Theframe-level confidence indicator is determined based on the stability ofthe signal power level of the reflected wave received by the radio-wavesensor within a frame. In some embodiments, the radio-wave sensor 602may be a pulse radar, a chirp radar, a frequency modulated continuouswave radar, but the present invention is not limited thereto.

The camera 604 captures and provides an image 612. The processor 606receives the spatial information 610 including the position and velocityof the foreground object, background and so on. In the scene, theprocessor 606 receives the image 612 captured by the camera 604, tracksat least one object/background based on the spatial information 610 andthe image 612 to obtain a consolidated tracking result. The processor606 predicts a motion trajectory of the object and background based onthe tracking result to obtain a prediction result, and controls thecamera 604 (for example, through a control signal 614) according to theprediction result. In some embodiments, the processor 606 executes AI/MLcontent detection 620 on the image 612 from the camera 604 to detectobjects in the image 612. In some embodiments, the processor 606executes EIS cropping 622 which reduces the image size to maintain thevisual stability based on the spatial information 610 and the image 612captured by the camera 604.

In some embodiments, tracking the at least one object based on thespatial information 610 and the image 612 by the processor 606 mayinclude processing the image 612 to obtain at least one property of theat least one object in the image 612; and fusing the spatial information610 with the at least one property of the at least one object in theimage 612 to obtain the tracking result. In some other embodiments,tracking the at least one object based on the spatial information 610and the image 612 by the processor 606 may include generating a firsttracking result based on the spatial information 610; generating asecond tracking result based on the image 612; and unifying the firsttracking result and the second tracking result to obtain theconsolidated tracking result. Specifically, the processor 606 setsweights of the first tracking result and the second tracking resultbased on at least one of the spatial information 610 and the image 612,and selects one of the first and second tracking result with the higherweight as the consolidated tracking result.

FIG. 6 is a flow chart of a method for EIS cropping in accordance withsome embodiments of the present invention. The method for EIS croppingof the present invention may include receiving spatial informationoutput by the radio-wave sensor 602, wherein the spatial information 610includes position and velocity of at least one point (step S600), orpatterns of motion; receiving an image 612 captured by the camera 604(step S602); determining, based on the spatial information 610 and theimage 612, that the motion of at least one object is local motion if themotion of the at least one object is inconsistent with most otherobjects (step S604); determining, based on the spatial information 610and the image 612, that the motion of the object is global motion if themotion of the at least one object is consistent with most other objects(step S606); filtering the global motion for EIS cropping (step S608);and providing the motion of the at least one object to processor 606 forEIS procession/cropping.

For example, when the processor 606 detects a large plane (such as awall) is moving up and down in a short period of time with a particularjittering pattern based on spatial information 610 of the target, theprocessor 606 determines that the moving of the large plane is unlikelypossible. Therefore, the processor determines that the large plane isthe global motion, so that the EIS cropping 622 filters the motion ofthe large plane.

In some embodiments, actuating and controlling the camera 604 with theprocessor 606 according to the final tracking result may include settingthe shutter timing, shutter speed and focal length of the camera basedon the result of motion prediction; and capturing and outputting a finalimage. In some other embodiments, actuating and controlling the camera604 using the processor 606 according to the final tracking result mayinclude setting the shutter speed, focal length, and exposure time delayof the camera based on the motion prediction, the expected position ofthe object is updated continuously in a prediction phase; and capturingand outputting an in-between image.

There are several advantages of the method and the electronic device 600of the present invention as follows, 1) inherent 4D/5D/6D point cloudand object tracking enables real-time accurate object detection,tracking and motion prediction; 2) able to capture the key moment(s) athigh quality with adaptive 3A control; 3) avoid hijacking,centralization, drifting and losing focus; 4) multiple points perobject, that can be independently tracked for better focus; 5) supportimage stabilization; 6) insensitive to color, lighting and otherenvironment impacts; 7) wide velocity detection range (high/low velocityare challenging for image processing); 8) wide FoV for better motionprediction; 9) long distance detection; 10) lower system (computing)power consumption; 11) fast processing speed; 12) small size and no holeopening for non-metallic material (for example, on a smart phone).

In advantage 2) described above, the 3A control means auto-focus,auto-exposure and auto white balance. In advantage 3) described above,the hijacking is a phenomenon that the tracking algorithm cannot telltwo objects with similar characteristics (e.g. having the same colors)when the two objects come close or cross each other. In advantage 3)described above, the centralization is a phenomenon, similar to thehijacking, that the tracking algorithm cannot tell two objects withsubstantially similar characteristics when the two objects are occludedor partially occluded. In advantage 3) described above, the drifting isa phenomenon the tracking results show a drift in time and space fromthe reality when the captured images or videos do not contain “keymoment(s)”.

For example, the steps S100˜S106 in FIG. 1 , the steps S304˜S310 in FIG.3A, the steps S304′˜S306′ in FIG. 3B can achieve effects of advantages1), 2), 3), 4), 6), 7), 8), 9) and 10) due to the physicalcharacteristics of the radio-wave sensor and the spatial informationobtained from the radio-wave sensor. Since the data size for processingspatial information from the radio-wave sensor is much less than thatfor image processing, advantage 11) can be achieved. Since the size ofchip packaging for radio-wave sensor is really small, advantage 12) canbe achieved. The steps S600˜S608 in FIG. 6 can achieve the effect ofadvantage 5) due to EIS procession/cropping with considering spatialinformation from the radio-wave sensor.

In the several embodiments provided by the present invention, it shouldbe understood that the disclosed system, device, and method can beimplemented using other methods. The device embodiments described aboveare merely illustrative, for example, the division of units is only alogical function division, and there may be other divisions in actualimplementation. For example, multiple units or elements can be combinedor integrated into another system, or some features may be omitted ornot implemented. In addition, the displayed or discussed mutual couplingor direct coupling or communicative connecting may be indirect couplingor communicatively connecting through some interfaces, device or units,and may be in electrical, mechanical, or other forms.

In addition, the functional units in the various embodiments of thepresent invention may be integrated into one processing unit, or eachunit may exist alone physically, or two or more units may be integratedinto one unit. The above-mentioned integrated unit can be realizedeither in the form of hardware or in the form of a software functionalunit.

Although the present invention is disclosed above in the preferredembodiment, it is not intended to limit the scope of the presentinvention. Anyone with ordinary knowledge in the relevant technicalfield can make changes and modifications without departing from thespirit and scope of the present invention. Therefore, the protectionscope of the present invention shall be determined by the scope of theclaims.

What is claimed is:
 1. A method for motion prediction, comprising:receiving spatial information output by a radio-wave sensor, wherein thespatial information comprises position and velocity of at least onepoint; receiving an image captured by a camera; tracking at least oneobject based on the spatial information and the image to obtain aconsolidated tracking result; predicting a motion trajectory of the atleast one object based on the consolidated tracking result to obtain aprediction result; and controlling the camera according to theprediction result.
 2. The method as claimed in claim 1, wherein thespatial information further comprises a frame-level confidenceindicator.
 3. The method as claimed in claim 2, wherein the frame-levelconfidence indicator is determined based on a stability of a signalpower level of a reflected wave received by the radio-wave sensor withina frame.
 4. The method as claimed in claim 1, wherein the step oftracking the at least one object based on the spatial information andthe image to obtain the consolidated tracking result comprises:processing the image to obtain at least one property of the at least oneobject in the image; fusing the spatial information with the at leastone property of the at least one object in the image to obtain theconsolidated tracking result.
 5. The method as claimed in claim 4,wherein the at least one property of the at least one object in theimage is obtained by processing the image using at least one ofArtificial Intelligence (AI), Machine Learning (ML) content detectionand computer vision (CV).
 6. The method as claimed in claim 1, whereinthe step of controlling the camera according to the prediction resultcomprises: setting at least one of camera parameters and imageparameters based on the prediction result; and capturing a final imagebased on the set camera parameters and the set image parameters.
 7. Themethod as claimed in claim 6, wherein the camera parameters comprise atleast one of a shutter timing, a shutter speed and a focal length of thecamera based on the prediction result; wherein the image parameters arededicated for image processing.
 8. The method as claimed in claim 1,wherein the step of tracking the at least one object based on thespatial information and the image to obtain the consolidated trackingresult comprises: generating a first tracking result based on thespatial information; generating a second tracking result based on theimage; and unifying the first tracking result and the second trackingresult to obtain the consolidated tracking result.
 9. The method asclaimed in claim 8, wherein the step of unifying the first trackingresult and the second tracking result comprises: setting weights of thefirst tracking result and the second tracking result based on at leastone of the spatial information and the image; and selecting one of thefirst and second tracking results with the higher weight as theconsolidated tracking result.
 10. A method for Electric ImageStabilization (EIS) cropping, comprising: receiving spatial informationoutput by a radio-wave sensor, wherein the spatial information comprisesposition and velocity of at least one point; receiving an image capturedby a camera; determining, based on the spatial information and theimage, that motion of at least one object is local motion if the motionof the at least one object is inconsistent with most other objects;determining, based on the spatial information and the image, that themotion of the at least one object is global motion if the motion of theat least one object is consistent with most other objects; and filteringthe global motion for EIS cropping.
 11. An electronic device,comprising: a radio-wave sensor, configured to provide spatialinformation including position and velocity of at least one point; acamera, configured to provide an image; a processor, configured toexecute the following steps: receiving the spatial information includingthe position and velocity of the at least one point; receiving the imagecaptured by the camera; tracking at least one object based on thespatial information and the image to obtain a consolidated trackingresult; predicting a motion trajectory of the at least one object basedon the tracking result to obtain a prediction result; and controllingthe camera according to the prediction result.
 12. The electronic deviceas claimed in claim 11, wherein the spatial information furthercomprises a frame-level confidence indicator; and the frame-levelconfidence indicator is determined based on a stability of a signalpower level of a reflected wave received by the radio-wave sensor withina frame.
 13. The electronic device as claimed in claim 11, wherein theprocessor is configured to track the at least one object based on thespatial information and the image to obtain the consolidated trackingresult, comprising: processing the image to obtain at least one propertyof the at least one object in the image; fusing the spatial informationwith the at least one property of the at least one object in the imageto obtain the consolidated tracking result.
 14. The electronic device asclaimed in claim 13, wherein the at least one property of the at leastone object in the image is obtained by processing the image using atleast one of Artificial Intelligence (AI), Machine Learning (ML) contentdetection and computer vision (CV).
 15. The electronic device as claimedin claim 11, wherein the processor is configured to track the at leastone object based on the spatial information and the image to obtain theconsolidated tracking result, comprising: generating a first trackingresult based on the spatial information; generating a second trackingresult based on the image; and unifying the first tracking result andthe second tracking result to obtain the consolidated tracking result.16. The electronic device as claimed in claim 15, wherein the processoris configured to unify the first tracking result and the second trackingresult to obtain the consolidated tracking result, comprising: settingweights of the first tracking result and the second tracking resultbased on at least one of the spatial information and the image; andselecting one of the first and second tracking results with the higherweight as the consolidated tracking result.
 17. The electronic device asclaimed in claim 11, wherein the processor is configured to control thecamera according to the prediction result, comprising: setting at leastone of camera parameters and image parameters based on the predictionresult; and capturing a final image based on the set camera parametersand the set image parameters.
 18. The electronic device as claimed inclaim 17, wherein the camera parameters comprise at least one of ashutter timing, a shutter speed and a focal length of the camera basedon the prediction result; wherein the image parameters are dedicated forimage processing.
 19. An electronic device for Electric ImageStabilization (EIS) cropping, comprising: a radio-wave sensor,configured to provide spatial information including position andvelocity of at least one point; a camera, configured to provide animage; a processor, configured to execute the following steps: receivingspatial information output by the radio-wave sensor; receiving the imagecaptured by the camera; determining, based on the spatial informationand the image, that motion of at least one object is local motion if themotion of the at least one object is inconsistent with most otherobjects; determining, based on the spatial information and the image,that the motion of the at least one object is global motion if themotion of the at least one object is consistent with most other objects;and filtering the global motion for EIS cropping.